Search for: All records

Recent innovations in virtual and mixed-reality (VR/MR) technologies have enabled innovative hands-on training applications in high-risk/high-value fields such as medicine, flight, and worker-safety. Here, we present a detailed description of a novel VR/MR tactile user interactions/interface (TUI) hardware and software development framework that enables the rapid and cost-effective no-code development, optimization, and distribution of fully authentic hands-on VR/MR laboratory training experiences in the physical and life sciences. We applied our framework to the development and optimization of an introductory pipette calibration activity that is often carried out in real chemistry and biochemistry labs. Our approach provides users with nuanced real-time feedback on both their psychomotor skills during data acquisition and their attention to detail when conducting data analysis procedures. The cost-effectiveness of our approach relative to traditional face-to-face science labs improves access to quality hands-on science lab experiences. Importantly, the no-code nature of this Hands-On Virtual-Reality (HOVR) Lab platform enables faculties to iteratively optimize VR/MR experiences to meet their student’s targeted needs without costly software development cycles. Our platform also accommodates TUIs using either standard virtual-reality controllers (VR TUI mode) or fully functional hand-held physical lab tools (MR TUI mode). In the latter case, physical lab tools are strategically retrofitted with optical tracking markers to enable tactile, experimental, and analytical authenticity scientific experimentation. Preliminary user study data highlights the strengths and weaknesses of our generalized approach regarding student affective and cognitive student learning outcomes. 

Convolutional Neural Networks (CNNs) have been explored to detect forced oscillations in windfarm systems in the past. However, these CNNs require a significant amount of data samples between inference queries and a significant amount of computational power and time. This leads to systems that have a large delay between a forced oscillation occurring and detecting the forced oscillation. This paper presents a novel approach applying Hyperdimensional Computing (HDC) as an effective solution for the first time in forced oscillation detection to overcome the problems of CNNs. HDC is able to reduce the time to detect forced oscillations in two ways: First, by reducing the time needed to collect data to create a new inference sample by reducing the number of data points required. Second, by providing a significantly smaller, more energy efficient, and faster model for detection than current state-of-the-art. Our results show that HDC, with an FPGA implementation, is able to achieve 55× faster detection of forced oscillations in windfarms while achieving the same accuracy as the best current CNN models using software solutions. 

Despite recent advances in deep learning, object detection and tracking still require considerable manual and computational effort. First, we need to collect and create a database of hundreds or thousands of images of the target objects. Next we must annotate or curate the images to indicate the presence and position of the target objects within those images. Finally, we must train a CNN (convolution neural network) model to detect and locate the target objects in new images. This training is usually computationally intensive, consists of thousands of epochs, and can take tens of hours for each target object. Even after the model training in completed, there is still a chance of failure if the real-time tracking and object detection phases lack sufficient accuracy, precision, and/or speed for many important applications. Here we present a system and approach which minimizes the computational expense of the various steps in the training and real-time tracking process outlined above of for applications in the development of mixed-reality science laboratory experiences by using non-intrusive object-encoding 2D QR codes that are mounted directly onto the surfaces of the lab tools to be tracked. This system can start detecting and tracking it immediately and eliminates the laborious process of acquiring and annotating a new training dataset for every new lab tool to be tracked. 

Active tracking enables higher precision in tracking the positions, orientations, and states of the virtualized objects. STEAMVR Lighthouse tracking base-stations can be used for tracking specific objects. However, current solutions are bulky and costly. The overall goal of this research work was to reduce the size and cost of active VR trackers to enable their attachment to ever smaller physical tools and objects to be tracked in the real world and displayed in a virtual reality environment.